Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide overrides for front matter input specifications #50

Merged
merged 4 commits into from
May 29, 2024
Merged

Conversation

mdales
Copy link
Collaborator

@mdales mdales commented May 28, 2024

In the past we have had the input key in the front matter definition for a sharkdown file:

---
inputs:
- "/data/test/1.csv"
- "/data/other/data.txt"
---
...

In this PR I do two things mostly:

  • That input map now has key/value pairs
  • You can specify overrides of the values for a key from the command line

This means that now your map would look like:

---
inputs:
  input1: "/data/test/1.csv"
  input2: "/data/other/data.txt"
---
...

And then you can invoke it with:

$ dune exec -- md --input1=/maps/mwd24/somedata.csv --input2=~/test/data.txt ...

This allows easier use of shark as a command line exploratory system in addition to being a more CI based thing. There's an interesting question of how you specify these inputs in a CI world - this kind of dynamic binding is somewhat the antithesis of what shark is trying to achieve (setting everything in stone for reproducibility reasons), but I think pragmatically we need something like this to deal with there exploratory science before you get the world set in stone.

To help with the traceability, this change will insert a new shark-import block into the markdown itself, so you can see what happened in the output markdown. This means if you have the results you can see what the substitutions were at least.

Additional changes by this PR:

  • The front matter imports now appear in dot
  • shark-import blocks appear in dot
  • Added simple http importer block

@mdales mdales requested a review from patricoferris May 28, 2024 13:18
Copy link
Collaborator

@patricoferris patricoferris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great thanks! Mostly little nits that can be ignored if you like :))

@mdales mdales merged commit f70b778 into main May 29, 2024
2 checks passed
@mdales mdales deleted the mwd-import branch May 29, 2024 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants